Automation of sequences of actions

ABSTRACT

Traditional manual macro-recorders may not work under a dynamically changing operating environment. Technical solutions are disclosed to automatically generate macros to increase productivity. After a new sequence of actions is detected, the system will prompt the user with the information of an existing macro if the existing macro contains a similar sequence. Otherwise, the system will attempt to automatically generate a new macro based on the sequence of actions.

BACKGROUND

Automation refers to technology to perform a process without humanassistance. Automation has been practiced in practically every kind ofmanufacturing process, particularly since the Industrial Revolution.Nowadays, computers usually serve as a critical component to controlautomated processes. Sometimes, operators interact with computer systemsin repetitive actions, which themselves may also be automated.

A macro-recorder is a piece of software to record a user's interactionswith computer systems. The recorded actions may be played back by theauthor or other users. In this way, many users can perform repeatable,complex actions without computer programming. By way of example, mostword processors have built-in macro-recorders to facilitate users toautomate user's actions to some extent.

Traditional macro-recorders may work well in a relatively staticenvironment. Users may control when and what to record, andmacro-recorders oftentimes just record mouse and keyboard functionswithout analyzing the operating environment for user actions. It isproblematic to play back such macros if the operating environmentchanges. Further, it is a challenge to train users to properly recordmacros in today's increasingly dynamic and collaborative computingenvironment with cloud computing.

SUMMARY

Embodiments of the present disclosure relate to systems and methods forautomation of sequences of actions. As such, one aspect of the presentdisclosure is to detect a sequence of actions related to a user on acomputing system. Another aspect of the present disclosure is to measurethe similarity of two sequences of actions. Yet another aspect of thepresent disclosure is to search macros similar to the detected sequenceof actions. Yet another aspect of the present disclosure is toautomatically generate a macro based at least in part on the detectedsequence of actions.

As described in embodiments herein, technical solutions are provided toautomatically generate macros to provide consistent quality and increaseproductivity for a process. In various embodiments, if a similarsequence of actions is found in a macro repository, the system willprompt the user with the information of the existing macro. The systemwill consider various characteristics of two sequences of actions forsimilarity measurement, such as the nature of the actions, thecomplexity of the actions, and the order of the sequence of actions.Otherwise, the system will attempt to automatically generate a new macrobased on the sequence of actions. To evaluate whether to generate a newmacro, the system will evaluate the sequence of actions for itsfrequency, duration, complexity, etc.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be readily understood by the following detaileddescription in conjunction with the accompanying drawings.

FIG. 1 is a schematic diagram illustrating an example implementation ofa system for automation of sequences of actions, incorporating aspectsof the present disclosure, in accordance with one embodiment.

FIG. 2 is a schematic diagram illustrating an example action and anexample sequence, incorporating aspects of the present disclosure, inaccordance with one embodiment.

FIG. 3 is a flow diagram of a first example process for automation of asequence of actions, incorporating aspects of the present disclosure, inaccordance with one embodiment.

FIG. 4 is a flow diagram of a second example process for automation of asequence of actions, incorporating aspects of the present disclosure, inaccordance with one embodiment.

FIG. 5 illustrates an example computing device suitable for practicingthe disclosed embodiments, in accordance with one embodiment.

DETAILED DESCRIPTION

Cloud computing is a modern computing model enabling ubiquitous accessto shared pools of resources, including computer networks, servers,storage, applications, and services. Resources are shared on a cloudcomputing platform. Applications and services may be dynamically updatedand rapidly provisioned with minimal management effort. Users maycollaborate in creating and accessing content. Unlike traditional,relatively static desktop computing environments, cloud computing relieson the dynamics of sharing resources to achieve coherence, consistency,and economy of scale.

Many user interactions with a computer system are repetitive.Traditional macro-recorders are designed to work in a relatively staticenvironment, e.g., the traditional desktop computing model. Users maycontrol when and what to record in a macro, so that the macro may beplayed back in the same computing environment. It is problematic torecord and play back such macros in a dynamically changing cloudcomputing environment. Further, it is a challenge to train users toproperly record macros in dynamic and collaborative computingenvironments. Even further, individually recorded macros cannot beeasily shared.

In this disclosure, technical solutions are provided to automatesequences of actions performed by users on a computing system. Thetechnical solutions disclosed herein are not limited to, but can beimplemented with the cloud computing, such as Adobe Marketing Cloud® ora content management system. Embodiments of the present disclosureinclude a system for automation of sequences of actions and relatedprocesses for automatically generating macros. To this end, sequences ofactions are detected by the system. If a sequence of action is qualifiedto form a new macro, the system will automatically generate the newmacro. The new macro may be shared or reused by other users.

At a high level, and as described in reference to illustrativeembodiments, actions are tagged with a complexity score or weight. Thesystem keeps track of sequences of actions associated with attributessuch as frequency, duration, and complexity. The system determines thequalification for a sequence to be persisted as a macro based onthresholds for various attributes such as the frequency, the duration,and the complexity of the sequence. If a sequence is qualified as acandidate for a new macro, the system may further solicit a validationfrom the user. If the user rejects the generation of the new macro, thesystem may adjust the thresholds for qualification accordingly, e.g.,increasing one or more thresholds. Otherwise, the system may generatethe new macro to be used by the same user or other users in the future.Further, as the system evolves and resources are dynamicallyreconfigured, macros may also be automatically updated by the system.

In one implementation, the system identifies sequences of actions oftenexecuted by a user. When a new sequence is identified and qualified tobecome a macro based on, e.g., the frequency, the proximity with knownsequences, the average duration, or the complexity of the sequence, anew macro may be generated with a label and a description based on theactions of the sequence. A keyboard shortcut is designated for the newlygenerated macro. Afterwards, the system may suggest the keyboardshortcut of an existing macro when it identifies that any user hasperformed or likely is performing a known sequence of actions includedin the existing macro. To accomplish that, the system will measure thesimilarity between the newly detected sequence of actions and theexisting sequence of actions in the macro, e.g., based on varioussimilarity criteria, which will be further discussed in connection withvarious figures herein.

In summary, the disclosed technical solutions provide variousadvantages. Users are no longer required to manually record macros fortheir individual workflow. Instead, the system automatically detects andgenerates macros. The system-generated macros are shared by other usersto reduce operation complexity and time, reduce labor costs andexpenses, increase system-wide throughput or productivity, and improveconsistency and quality of processes or work products. Further, thesystem may update macros as the system evolves and resources aredynamically reconfigured.

Various terms are used throughout this description. Definitions of someterms are included below to provide a clearer understanding of the ideasdisclosed herein.

The term “action” or “user action” refers to any operation performed bya user to interact with the computing system. User actions are meant totrigger the execution of software codes via an act of the user, such asclicking on a link or button via a user interface interaction, issuing acommand via a command console, etc. Actions usually are predefined, suchas standard action types defined by system developers for the most basicuse cases.

The term “sequence” refers to a sequence of actions. A sequence is achain of multiple actions, which result in a change in the system, suchas a database update, either local or remote. A sequence usuallyinvolves an order of the involved actions. Accordingly, two sequencesmay be differentiated not only based on the actions involved, but alsothe order of the involved actions.

The term “macro” generally refers to a rule or pattern that specifieshow a certain input sequence (e.g., a keyboard shortcut, or a menu item)should be mapped to a replacement output sequence (e.g., a sequence ofactions). As used herein, the term “macro” specifically refers toreusable sequences of actions that are recognized by the computersystem. A macro includes at least one sequence of actions. In someembodiments, a macro includes two or more sequences of actions. Eachsequence of action can independently perform a function, and allsequences of actions in a macro can collectively perform a more complexfunction. A macro has its title, description, and many other metadatafields.

Referring now to FIG. 1, a schematic diagram illustrates an exampleimplementation of a system for automation of sequences of actions. Inaccordance with various embodiments, users 150 interact with system 100via various user actions. As illustrated in FIG. 1, the system forautomation of sequences of actions includes sequence monitor 110, actionmanager 120, macro generator 130, and macro manager 140, operativelycoupled with each other.

Action manager 120 enables system 100 to store, retrieve, search, orupdate properties of actions. Various properties of actions may be usedto verify whether a form of user interaction with the system is arecognizable action. Actions are tagged with a complexity score in someembodiments. Actions have labels and many other properties, which willbe further discussed in connection with FIG. 2. Actions and theirproperties usually are predefined by system developers for common usecases, such as standard action types defined for editing an image in acontent management system. Actions and their properties may also bedefined by system developers or users for customized use cases for aspecific user or a group of users.

Sequence monitor 110 detects actions performed by a user. User actionstrigger software codes to be executed either locally on the user'scomputer or remotely on a remote computer. In some embodiments, sequencemonitor 110 has a local agent installed on the client's computer tomonitor each action in real time. In some embodiments, sequence monitor110 will detect and analyze the involved actions whenever the softwarecodes on the remote computer are called to execute.

Sequence monitor 110 will inquire action manager 120 to verify whetherthe characteristic of the invoked software codes meets the predefinedcharacteristics of an action. Not every user interaction will beconfirmed as a recognizable action by action manager 120. By way ofexample, an accidental user interaction that triggered the return of anerror code will not be confirmed as a valid user action by actionmanager 120.

However, if an action is confirmed, sequence monitor 110 will furtheranalyze the relationship of the newly detected action and any actionsdetected previously to determine whether two sequential actions may beput together into a temporary sequence. The newly detected action may bethe beginning of a new sequence or a continuance of the ongoingsequence. Sequence monitor 110 has predefined rules to delineate whethertwo actions may be executed sequentially because some actions cannot bemixed in a sequence. In one embodiment, if two sequential actions cannotbe linked for a common task, sequence monitor 110 will treat the mostrecent action as the beginning of a new sequence. Sequence monitor 110may look at the timing of two sequential actions. In one embodiment, ifthe time lag between two actions is greater than a predefined threshold,sequence monitor 110 will treat the most recent action as the beginningof a new sequence. Sequence monitor 110 may analyze the user identitiesassociated with two sequential actions. In one embodiment, if the useridentities change between the two sequential actions, sequence monitor110 will not treat the two sequential actions as one sequence. Inanother embodiment, sequence monitor 110 will treat actions performed bycollaborative users as a multi-user sequence. In various embodiments,sequence monitor 110 may determine a sequence based on various otherrules.

Once a sequence is detected, sequence monitor 110 will identify variousproperties associated with the sequence, such as the frequency of thesequence being performed by the same user or other users, the durationof the sequence, the complexity of the sequence, the location ofperforming the sequence, the collaborative users involved in thesequence, the project associated with the sequence, the user preferenceof the user who performed the sequence, and so forth. These and otherproperties of sequences are further discussed in connection with FIG. 3.

A sequence requires two or more actions. Whenever a newly confirmedaction is added to the sequence, sequence monitor 110 will update theproperties of the sequence. Subsequently, sequence monitor 110 queriesmacro manager 140 whether the sequence matches any existing macro. Macromanager 140 compares the sequence with another sequence defined in anexisting macro based on various similarity measurements. In someembodiments, macro manager 140 compares two sequences based at least inpart on their cosine similarity score. In some embodiments, macromanager 140 compares two sequences based at least in part on their editdistance score. In various embodiments, macro manager 140 considers thevarious properties of actions (e.g., complexity) and the variousproperties of the sequences (e.g., the order of actions) in evaluatingsimilarity between two sequences.

If an existing macro with a similar sequence is identified, macromanager 140 may prompt the user with information of the existing macro,such as the description of the macro and the keyboard shortcut of themacro. In this way, the user may use the keyboard shortcut to performthe whole sequence of actions in the future. In one embodiment, macromanager 140 performs a partial match between a newly detected sequenceand an existing macro. In this way, macros having similar sequences ofactions in the beginning, middle, or end of the macro may all bereturned as candidate macros for the user to choose.

However, if no existing macro is found, or the user requests to create anew macro, macro generator 130 will then determine whether the sequenceis qualified to be recorded as a new macro. To this end, macro generator130 determines the qualification for a sequence to be persisted as amacro based on thresholds for various properties of the sequence, suchas the frequency, the duration, and the complexity of the sequence.

In one embodiment, if a sequence is qualified as a candidate for a newmacro, macro generator 130 may nonetheless solicit a validation from theuser. If the user rejects the generation of the new macro, macrogenerator 130 may adjust the thresholds for qualification accordingly,e.g., increasing one or more thresholds. By way of example, the user mayreject to generate a new macro because the user believes that thesequence has been used infrequently. Macro generator 130, in this case,may heuristically increase the frequency requirement for generating anew macro for the user.

Otherwise, if the user approves the proposal to generate the new macro,macro generator 130 will generate the new macro based on the sequence.Macro generator 130 may assign a label and a description to the newmacro based on the properties of the sequence and the properties ofactions in the sequence. Macro generator 130 may assign the macro into amenu that may be invoked via a user interface to prompt users to usemacros. Further, macro generator 130 may assign a keyboard shortcut tothe newly generated macro so the user may invoke the macro quickly. Theprocess of generating a new macro is further discussed in connectionwith FIG. 3 and FIG. 4.

In some embodiments, macro manager 140 may allow the macros generated bymacro generator 130 to be used by any users who may be benefited to usesuch macros to improve productivity. In other embodiments, macro manager140 may limit the access privilege of a macro to the original user orusers related to the original user in a group. Users may be allowed toset the access privilege during or after the generation of the macro.

Traditionally, a sequence of actions is performed by a single user.System 100, as disclosed herein, monitors actions from users 150 and canautomatically generate macros from multi-user sequences. In acollaborative environment, multi-user sequences usually have highercomplexity compared to traditional single-user sequences. Users involvedin a multi-user sequence may have different access privileges toresources. One user's action may be conditioned on another user'saction. Resultantly, the order of actions in a multi-user sequence maybe critical to secure the execution of the multi-user sequence. System100 may capture the dynamics of multi-user sequences and encode useridentities and privileges into a macro. When the multi-user macro isplayed back, macro manager 140 may enable each action to be executedwith the same privilege inherited from the original user.

Further, as the system evolves, users change, or resources arereconfigured, macro manager 140 may dynamically update existing macros.By way of example only, if an action in a macro becomes unavailable,macro manager 140 may substitute the unavailable action with anotheraction with equivalent function. Similarly, if a piece of resourcebecomes unavailable, macro manager 140 may substitute it with anotherpiece of equivalent resource. When a piece of software is updated, macromanager 140 may have to adjust the order to execute the sequence ofactions in a macro accordingly. In some embodiments, macro manager 140may replace a stale macro with another newly formed macro if theyperform substantially the same function. Macro manager 140 may alsocombine multiple existing macros to form a new macro for comprehensivefunctions. In general, macro manager 140 uses advantages of cloudcomputing to dynamically reconfigure macros as the computing environmentchanges.

In some embodiments, system 100 is a server computing device and/orservice, such as a server and/or a service provided in a computingcloud, and interacts with other servers and/or services as well asvarious users. In alternative embodiments, system 100 or parts of system100 can be integrated with a user computing device as a piece ofhardware or software, such as embodied as a plug-in or add-on to abrowser. In some embodiments, system 100 is embodied as a specializedcomputing device. In some embodiments, system 100 can be embodied, forexample, as an application, a mobile application, or an online computingservice. In some embodiments, system 100 can be a distributed system,for example, each of sequence monitor 110, action manager 120, macrogenerator 130, and macro manager 140 can be distributed across anynumber of servers. Regardless of the computing platform on which system100 is implemented, system 100 can be embodied as a hardware component,a software component, or any combination thereof for automation ofsequences of actions of digitized forms.

In other embodiments, systems for automation of sequences of actions canbe implemented differently than that depicted in FIG. 1. As an example,macro generator 130 can be combined with macro manager 140 to form acomprehensive component to generate and manage macros. In someembodiments, components depicted in FIG. 1 have a direct or indirectconnection not shown in FIG. 1. In this way, the components depicted inFIG. 1 can be connected in any number of ways and are not limited to theconnections illustrated in FIG. 1. In some embodiments, some of thecomponents depicted in FIG. 1 are divided into multiple components.Further, one or more components of system 100 can be located across anynumber of different devices and/or networks. As an example, actionmanager 120 or macro manager 140 can be implemented as an independentcomponent in a computing cloud.

FIG. 2 is a schematic diagram illustrating an example action and anexample sequence, in accordance with an embodiment of the presentdisclosure. Action 210 is an operation associated with a user'sinteraction with system 100 in FIG. 1, such as clicking on a link orbutton via a user interface interaction, issuing a command via a commandconsole, etc. Action 210 causes the execution of software codes orchanges the status of an entity in the computing system.

Sequence 220 is composed of two or more actions, such as action 210. Theexecution of the chain of actions in sequence 220 will result in achange in the system, such as a status change of an item in the system,e.g., via a database update. Actions in sequence 220 may collectivelyaccomplish a function. A sequence usually has a defined order of theinvolved actions. Accordingly, two sequences may be differentiated notonly based on the actions involved, but also the order of the involvedactions. In general, the order of actions in a sequence is important.However, in some embodiments, the order of actions does not affect thefunction accomplished by the sequence.

Action 210 and its properties, including property 212 and property 216,usually are predefined by system developers for common use cases forgeneral users, or customized use cases for a specific user or a group ofusers. In one embodiment, action 210 is an operation for photo editing.Property 212 is TOOL, and value 214 is the name of the tool, forinstance, rectangular marquee tool, move tool, polygon lasso tool, magicwand tool, crop tool, slice tool, brush tool, clone stamp tool, erasertool, blur tool, dodge tool, notes tool, zoom tool, etc. Property 216 isWEIGHT, and value 218 is the value of the weight assigned to respectivetools.

The weight of an action refers to the complexity of the action. The morecomplex the action, the higher its weight. The weight of an action isgenerally assigned by the system, and can be adjusted by users. Thecomplexity of an action refers to the computational complexity in someembodiments. Thus, if an action takes up more computational resources,the system will assign more weight to the action. The complexity of anaction refers to the time complexity for users in some embodiments.Thus, if an action will take users more time to complete, the systemwill assign more weight to the action. The complexity of an action mayrefer to other aspects in executing the action in other embodiments. Insome embodiments, the weight associated with an action may be learned bythe system, e.g., via monitoring how the action is executed by users.For example, the system may deduce the weight from the average timeusers spend on the action. Additionally, although not shown, action 210may have other properties and corresponding values, such as the brushsize for brush tool. In general, the properties and their associatedvalues define the characteristics of an action.

The characteristics of an action will be used, e.g., by sequence monitor110 or action manager 120, to verify whether the characteristics of adetected operation match the characteristics of a known action. Notevery user operation will be confirmed as an recognizable action byaction manager 120. For example, some user operations, e.g., clicking ona vacant area, may be rendered as moot. Some user operations, e.g.,issuing an unacceptable command, may trigger an error code. In variousembodiments, only when the detected operation is recognized by actionmanager 120 as one of the existing actions, sequence monitor 110 willadd the action to a sequence. Various characteristics of the action willthen affect the characteristics of the sequence.

Sequence monitor 110 will identify various properties associated withsequence 220. In one embodiment, property 222 is WEIGHT, and value 224is the value of weight, which refers to the overall complexity ofsequence 220. The overall complexity of a sequence may be the sum ofcomplexity scores of every action in the sequence, the averagecomplexity score of all actions in the sequence, the average complexityof unique actions in the sequence, the complexity of the primary actionin the sequence, the sum of complexity scores of the first three actionsin the sequence, and so forth depending on the implementation.

Property 226 and value 228 may represent the frequency of the sequencebeing performed by the same user or other users in some embodiments. Itwill be unnecessary to generate a new macro if the sequence is firstencountered by sequence monitor 110. However, if sequence monitor 110has detected the same sequence multiple times, or more than thefrequency threshold set in the system, then sequence monitor 110 willattempt to generate a new macro based on the sequence. The frequencythreshold may be set for a single user, a group, or all users in adomain. Accordingly, the macro may be published to the single user, thegroup, or all users in the domain.

Property 226 and value 228 may represent the duration of the sequence insome embodiments. The duration refers to the execution time of thesequence. If a user can rather quickly complete a sequence in a fewseconds, it may be counter-productive to force the user to remember anew macro for the sequence. However, if the duration exceeds theduration threshold, e.g., 10 minutes, then sequence monitor 110 mayqualify the sequence to become a new macro. In general, the durationthreshold for a multi-user sequence is longer than the durationthreshold for a single user. The duration threshold may be setdifferently for different users.

Additionally, although not shown, sequence 220 may have other propertiesand corresponding values, such as the location of performing thesequence, the resources required to execute the sequence, thecollaborative users involved in the sequence, the project associatedwith the sequence, the user preference of the user who performed thesequence, and so forth depending on the actual implementation. Ingeneral, the properties and their associated values define thecharacteristics of the sequence.

FIG. 3 is a flow diagram of a first example process 300 for automationof a sequence of actions, incorporating aspects of the presentdisclosure. In this example process, multiple actions may form aqualified sequence. Similar sequences in a repository will be searchedbased on the qualified sequence. Further, the sequence will be checkedfor its qualification to become a new macro. If the sequence passes suchqualification scrutiny, a new macro may be generated based on thesequence.

Action 310 is identified, e.g., by action manager 120 as a recognizableaction in the system. At block 320, sequence monitor 110 will determinewhether action 310 may be added to any monitored sequences. In someembodiments, multiple temporary sequences are being monitored bysequence monitor 110, for instance, for individual users, for differentgroups, for all users in a domain, etc. Action 310 may be added to oneor many temporary sequences. Generally, actions in a sequence arecoherent, and two sequential actions should not have conflict. Sequencemonitor 110 will conduct a conflict check before adding an action to atemporary sequence in some embodiments. However, if action 310 is notqualified for any temporary sequences, sequence monitor 110 may create anew temporary sequence, starting from action 310. Alternatively, thisprocess may exit.

In some embodiments, action 310 may be added to a temporary sequencebased on the user identity associated with action 310. For example,action 310 can be added to the temporary sequence created for the sameuser or the group including the user identity. In some embodiments,action 310 may be added to a temporary sequence based on the timing,location, or another characteristic of action 310. For example, action310 can be added to the temporary sequence whose most recent action isclose to the timing, location, or the other characteristic of action310.

After a sequence is added with action 310, sequence monitor 110 willdetermine whether an existing macro could be found to match with thenewly updated sequence at block 330. Two sequences may be compared basedat least in part on their cosine similarity score or their edit distancescore, as further discussed in FIG. 4. In various embodiments,properties of actions in a sequence and various properties of thesequences (e.g., the order of actions) may be used to evaluatesimilarities between two sequences.

If an existing macro with a similar sequence is identified, informationof the matching macro or macros, such as the description of the macroand the keyboard shortcut of the macro, is provided to the user at block340. In this way, the user may use the macro if the user confirms themacro accomplishes the function of the sequence.

However, if no matching macro is found, it will be determined at block350 whether the sequence is qualified to be recorded as a new macro.Thresholds associated with various properties of the sequence, such asthe frequency, the duration, and the complexity of a sequence, may beused to determine the qualification for a sequence to be persisted as amacro. To this end, at block 350, the system will determine whether oneor more characteristics of the sequence meet respective thresholds forcreating new macros. The characteristics of the sequence may include afrequency measurement of the sequence, a duration measurement of thesequence, or a complexity measurement of the sequence. Further,different thresholds may be set up for different sequences, such as forsingle-user sequences or multi-user sequences. Different thresholds maybe set up for different users. A threshold may also be automaticallyadjusted, e.g., in connection with block 390.

The sequence may be preprocessed into a new macro at block 360 if it isqualified as a candidate for a new macro. The preprocessing processinvolves prefilling one or more data fields in the macro based at leastin part on information of the sequence, such as the properties of thesequence and the properties of the actions in the sequence. By way ofexample, the preliminary values for the macro title and description maybe deduced from the properties of the sequence. Further, a preliminarykeyboard shortcut may be proposed by the system based on the propertiesof the sequence and existing keyboard shortcuts for existing macros.

With the preprocessed information of the macro, such as theauto-generated title, description, and keyboard shortcut, the user willbe prompted to validate the new macro at block 370. The user now has achance to review the proposed macro. The user may adjust the propertiesof the macro, such as the title, description, or keyboard shortcut. Theuser may even adjust the actions in the sequence, such as adding,removing, or adjusting the order of an action. In some embodiments, theprocess may skip block 360 and directly move to block 380 for generatingthe new macro.

After the user approves the proposal, the new macro may be generated atblock 380. The macro generator may assign the final label anddescription to the new macro based on the properties of the sequence andthe properties of actions in the sequence. The macro generator mayassign the macro into a menu, which may be invoked via a user interfaceto prompt users to use macros. The macro generator may assign a keyboardshortcut to the newly generated macro so that a user may invoke themacro quickly.

However, if the user rejects the proposal to generate a new macro, atblock 390, the system will adjust one or more thresholds for qualifyinga sequence to become a macro. A user may specifically indicate thereason for rejection at block 370, such as because the low frequency ofusage of the sequence, the low complexity of the sequence, etc.Accordingly, the system may adjust these thresholds, such as increasingthe frequency requirement for generating a new macro for the user.

FIG. 4 is a flow diagram of a second example process for automation of asequence of actions, incorporating aspects of the present disclosure.Process 400 can be performed, for example, by a system for automation ofsequences of actions, e.g., system 100 of FIG. 1. Process 400 can beperformed by processing logic that comprises hardware (e.g., circuitry,dedicated logic, programmable logic, microcode, etc.), software (e.g.,instructions run on a processing device to perform hardware simulation),or a combination thereof. The processing logic can be configured toautomate sequences of actions as macros. It will also be appreciatedthat, in various embodiments, process 400 can have fewer or additionaloperations than those depicted, or perform some of the depictedoperations in a different order without departing from the scope of thisdisclosure.

In various embodiments, the process begins at block 410, where asequence of actions is detected, e.g., enabled by sequence monitor 110of FIG. 1. A sequence includes two or more actions. When a useroperation is detected, action manager 120 will verify whether thedetected operation is a recognizable action in the system. To accomplishthis, action manager 120 can compare the characteristics of the detecteduser operation with the characteristics of the known action. Thecharacteristics of the detected user operation may include the identityof the user, the timestamp, the location, or the nature of the operation(e.g., which button is being clicked).

Once the user operation is determined to be an action, sequence monitor110 will either add the newly detected action to an existing pendingsequence or build a new sequence from the newly detected action. Whenmultiple pending sequences are being monitored for individual users ordifferent groups, sequence monitor 110 may add the newly detected actionto one or more pending sequences, e.g., based on the relationshipbetween the newly detected action and the pending sequences. However, ifthe newly detected action is incompatible with any pending sequences,sequence monitor 110 may create a new sequence, e.g., starting from thenewly detected action.

After the newly detected action is added to a sequence, sequence monitor110 will update the properties of the sequence. In some embodiments,sequence monitor 110 will determine the order of actions in thesequence, e.g., based on the timestamps associated with the actions, orthe natural order for adding the actions. Sequence monitor 110 mayretrieve various properties of involved actions and determinecorresponding properties for the sequence. By way of example, the sum ofcomplexity or the average complexity of the sequence may be determinedbased on the complexity score of each action. The complexity score orweight of an action may be predetermined and stored as a property of theaction. In some embodiments, the complexity score of an action may bedynamically determined, e.g., based at least in part on a duration toperform the action by the user or another user. By way of example, thecomplexity score may be set to be positively correlated with theduration. For instance, the longer it takes for the user to complete theaction, the higher a complexity score may be assigned to the action.Other properties of the sequence may also be determined at this stage.

At block 420, a macro repository may be searched for macros with similarsequences of actions as the newly formed sequence, e.g., by macromanager 140 of FIG. 1. Different search mechanisms may be used indifferent embodiments. A particular search mechanism may use one or moresimilarity measurements to compare two sequences based at least in parton a characteristic of the sequence to be searched, which includes theactions in the sequence, the order of the actions, the complexity of thesequence, the primary action in the action, etc.

In some embodiments, a complete match is required. In this case, theactions and their order in two sequences have to match completely.Further restrictions may be required in some embodiments, such as theuser privilege match, the user group match, etc., in addition to a matchfor the actions. In some embodiments, to accelerate the search process,the initial search may be based at least in part on the sum ofcomplexity scores or the average complexity score of all actions in thesequence. If the sum of complexity scores or the average complexityscore of all actions in a sequence deviates significantly from thesequence to be searched, they will unlikely be a complete match.

In some embodiments, an inclusion match is required. In this case, onesequence is required to contain the other sequence. If the firstsequence contains every action in the second sequence, the firstsequence is said to contain the second sequence. The order of actionsmay or may not matter. In one embodiment, the first sequence mustcontain the second sequence with the same order of actions.

In some embodiments, a primary action is identified for a sequence andsaved as a property of the sequence. The primary action usually is theaction with the highest complexity. The search process may be based atleast in part on the primary action. By way of example, the searchprocess may exclude all sequences without the primary action from thesearch space. In some embodiments, the similarity between two sequencesis determined based on their cosine similarity score. The cosinesimilarity score is the cosine of the angle between two non-zero vectorsof a vector space. Each action is notionally assigned a differentdimension. A sequence is characterized by a vector with differentactions as different dimensions in the vector space. The value of eachdimension corresponds in part to the number of times that action appearsin the sequence in some embodiments. In some embodiments, the similaritybetween two sequences is determined based at least in part on thecomplexity of actions in the sequence to be searched. In this case, thevalue of each dimension in the vector space also corresponds in part tothe complexity score of the action. Accordingly, the cosine similarityscore provides a useful measure of how similar two sequences are likelyto be in terms of the involved actions.

In some embodiments, the similarity between two sequences is determinedbased on their edit distance. The edit distance, as used herein, is toquantify how similar two sequences are to one another by counting theminimum number of operations required to transform one sequence into theother. A shorter edit distance denotes that two sequences are closerbecause it takes fewer operations to transform one sequence into theother.

In various embodiments, the similarity between two sequences isdetermined based at least in part on the order of actions in thesequence to be searched. By way of example, the similarity measurementbased on edit distance inherently considers the order of actions in asequence. In other similarity measures, the order of actions in asequence may be specifically emphasized in computing the finalsimilarity score. By way of example, the cosine similarity measurementdoes not directly consider the order of actions. Thus, in oneembodiment, the cosine similarity measurement is used in conjunctionwith the edit distance measure. The top rated sequences with high cosinesimilarity scores may be selected as initial candidates; these top ratedsequences may be re-ranked based on the edit distance score. In thisway, the process may reduce the computational complexity and considerboth similarity measures in a final mixed similarity score.

If a macro is found to contain the sequence to be searched, such as ifthe similarity measurement between the sequence in a macro and thesequence to be searched is above a similarity threshold, the system maysubsequently provide information of the macro to the user. Otherwise,the process may move to block 430 to automatically generate a macro.

Next, at block 430, a new macro may be generated for the sequence, e.g.,by macro generator 130 of FIG. 1. Macro generator 130 can deduce variousmetadata based at least in part on information of the sequence, such asthe properties of the sequence and the properties of the actions in thesequence. By way of example, the title of the sequence may be used asthe title of the macro. The primary action of the sequence may beemphasized in the description of the macro.

In various embodiments, macro generator 130 will optimize the sequenceto be encoded into the macro. By way of example, the user may have useda brush tool to paint an area multiple times until the area is filled upwith the selected color or pattern. Macro generator 130 may synthesizethese similar actions into one action to fill up the area with theselected color or pattern. Further, based on some predefined rules,macro generator 130 may optimize the actions in the sequence, such asaddition, deletion, or reorganization. By way of example, if action Pshould always be followed by action Q, but the sequence misses Q. Inthis case, macro generator 130 will insert action Q into the sequence.As another example, macro generator 130 may identify a portion of thesequence that is functionally equivalent to another macro, and thatmacro has other advantages. In this case, macro generator 130 maysubstitute the portion of the sequence with the sequence in that macro.In this way, the automatically generated macro may be optimized to anextent.

Having briefly described an overview of embodiments of the presentinvention, an exemplary operating environment in which embodiments ofthe present invention is to be implemented is described below in orderto provide a general context for various aspects of the presentinvention. Referring initially to FIG. 5 in particular, an exemplaryoperating environment for implementing embodiments of the presentinvention is shown and designated generally as computing device 500.Computing device 500 is but one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the invention. Neither should thecomputing device 500 be interpreted as having any dependency orrequirement relating to any one or combination of componentsillustrated.

The disclosure is described in the general context of computer code ormachine-useable instructions, including computer-executable instructionssuch as program modules, being executed by a computer or other machine,such as a personal data assistant or other handheld device. Generally,program modules including routines, programs, objects, components, datastructures, etc., refer to code that perform particular tasks orimplement particular abstract data types. The embodiments of thisdisclosure are to be practiced in a variety of system configurations,including handheld devices, consumer electronics, general-purposecomputers, more specialty computing devices, etc. The embodiments ofthis disclosure are to be practiced in distributed computingenvironments where tasks are performed by remote-processing devices thatare linked through a communications network.

With reference to FIG. 5, computing device 500 includes a bus 510 thatdirectly or indirectly couples the following devices: memory 520, one ormore processors 530, one or more presentation components 540,input/output (I/O) ports 550, input/output (I/O) components 560, and anillustrative power supply 570. Bus 510 represents one or more busses(such as an address bus, data bus, or combination thereof). Although thevarious blocks of FIG. 5 are shown with lines for the sake of clarity,in reality, delineating various components is not so clear, andmetaphorically, the lines would more accurately be fuzzy. For example, apresentation component such as a display device could also be consideredas an I/O component. Also, processors have memory. The inventorrecognizes that such is the nature of the art, and reiterates that thediagram of FIG. 5 is merely illustrative of an exemplary computingdevice that is used in connection with one or more embodiments of thepresent invention. Distinction is not made between such categories as“workstation,” “server,” “laptop,” “handheld device,” etc., as all arecontemplated within the scope of FIG. 5 and reference to “computingdevice.”

Computing device 500 typically includes a variety of computer-readablemedia. Computer-readable media include any available media to beaccessed by computing device 500, and include both volatile andnonvolatile media, and removable and non-removable media. By way ofexample, and not limitation, computer-readable media comprises computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules, orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which is used to store the desired information andwhich is accessed by computing device 500. Computer storage media doesnot comprise signals per se. Communication media typically embodiescomputer-readable instructions, data structures, program modules orother data in a modulated data signal such as a carrier wave or othertransport mechanism and includes any information delivery media. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared, and otherwireless media. Combinations of any of the above should also be includedwithin the scope of computer-readable media.

Memory 520 includes computer storage media in the form of volatileand/or nonvolatile memory. In various embodiments, the memory isremovable, non-removable, or a combination thereof. Exemplary hardwaredevices include solid-state memory, hard drives, optical-disc drives,etc. Computing device 500 includes one or more processors 530 that readdata from various entities such as memory 520 or I/O components 560.Presentation component(s) 540 present data indications to a user orother device. Exemplary presentation components include a displaydevice, speaker, printing component, vibrating component, etc.

In various embodiments, memory 520 includes, in particular, temporal andpersistent copies of macro logic 522. Macro logic 522 includesinstructions that, when executed by one or more processors 530, resultin computing device 500 to evaluate conversion quality of a form or aform object, such as, but not limited to, process 300 or process 400. Invarious embodiments, macro logic 522 includes instructions that, whenexecuted by processors 530, result in computing device 500 performingvarious functions associated with, but not limited to, sequence monitor110, action manager 120, macro generator 130, or macro manager 140, inconnection with FIG. 1.

In some embodiments, one or more processors 530 are to be packagedtogether with macro logic 522. In some embodiments, one or moreprocessors 530 are to be packaged together with macro logic 522 to forma System in Package (SiP). In some embodiments, one or more processors530 are integrated on the same die with macro logic 522. In someembodiments, processors 530 are integrated on the same die with macrologic 522 to form a System on Chip (SoC).

I/O ports 550 allow computing device 500 to be logically coupled toother devices including I/O components 560, some of which are built-incomponents. Illustrative components include a microphone, joystick, gamepad, satellite dish, scanner, printer, wireless device, etc. In someembodiments, the I/O components 560 also provide a natural userinterface (NUI) that processes air gestures, voice, or otherphysiological inputs generated by a user. In some embodiments, inputsare to be transmitted to an appropriate network element for furtherprocessing. An NUI implements any combination of speech recognition,stylus recognition, facial recognition, biometric recognition, gesturerecognition both on screen and adjacent to the screen, air gestures,head and eye tracking, and touch recognition (as described in moredetail below) associated with a display of the computing device 500. Thecomputing device 500 is equipped with depth cameras, such asstereoscopic camera systems, infrared camera systems, RGB camerasystems, touchscreen technology, and combinations of these, for gesturedetection and recognition. Additionally, the computing device 500 isequipped with accelerometers or gyroscopes that enable detection ofmotion. The output of the accelerometers or gyroscopes is to be providedto the display of the computing device 500 to render immersive augmentedreality or virtual reality.

Although certain embodiments have been illustrated and described hereinfor purposes of description, a wide variety of alternate and/orequivalent embodiments or implementations calculated to achieve the samepurposes could be substituted for the embodiments shown and describedwithout departing from the scope of the present disclosure. Thisapplication is intended to cover any adaptations or variations of theembodiments discussed herein. Therefore, it is manifestly intended thatembodiments described herein be limited only by the claims.

What is claimed is:
 1. A computer-implemented method, comprising:detecting, by a processor, a first sequence of actions related to a useron a computing system; searching, by the processor, a second sequence ofactions from a repository for sequences of actions based on the firstsequence of actions; and in response to a similarity measurement betweenthe first sequence of actions and the second sequence of actions beingbelow a similarity threshold, automatically generating a macro based onthe first sequence of actions.
 2. The method of claim 1, furthercomprising: in response to a similarity measurement between the firstsequence of actions and the second sequence of actions not being belowthe threshold, providing information of the second sequence of actionsto the user.
 3. The method of claim 1, wherein detecting the firstsequence of actions comprises: determining an order of the firstsequence of actions; retrieving complexity scores of respective actionsin the first sequence of actions; and calculating a sum of complexityscores or an average complexity score of the first sequence of actionsbased on the complexity scores.
 4. The method of claim 3, whereinsearching the second sequence of actions comprises searching therepository based at least in part on the sum of complexity scores or anaverage complexity score of the first sequence of actions.
 5. The methodof claim 3, wherein searching the second sequence of actions comprisesidentifying a primary action among the first sequence of actions basedat least in part on the complexity score of each action in the firstsequence of actions; and searching the repository based at least in parton the primary action.
 6. The method of claim 1, wherein searching thesecond sequence of actions comprises: measuring a similarity between thefirst sequence of actions and the second sequence of actions based atleast in part on respective complexity scores assigned to actions in thefirst sequence of actions and the second sequence of actions.
 7. Themethod of claim 1, wherein searching the second sequence of actionscomprises: measuring a similarity between the first sequence of actionsand the second sequence of actions based at least in part on an editingdistance to transform the first sequence of actions into the secondsequence of actions.
 8. The method of claim 1, wherein generating themacro comprises: determining whether a plurality of characteristics ofthe first sequence of actions meet respective thresholds for creatingnew macros.
 9. The method of claim 8, further comprising: providing avalidation request to the user to validate the macro; and in response toa user's negative input to the validation request, adjusting a thresholdfor creating new macros.
 10. The method of claim 8, wherein theplurality of characteristics of the first sequence of actions comprisesa frequency measurement of the first sequence of actions, a durationmeasurement of the first sequence of actions, or a complexitymeasurement of the first sequence of actions.
 11. The method of claim 1,wherein generating the macro comprises: prefilling one or more datafields in the macro based at least in part on information of the firstsequence of actions.
 12. One or more computer storage media comprisingcomputer-implemented instructions that, when used by one or morecomputing devices, cause the one or more computing devices to: detect afirst sequence of actions related to a user on a computing system;search a repository of sequences of actions for a second sequence ofactions that is similar to the first sequence of actions based at leastin part on a characteristic of the first sequence of actions; determinea similarity measurement between the first sequence of actions and thesecond sequence of actions; and automatically generate a macro based atleast in part on the first sequence of actions and the similaritymeasurement.
 13. The one or more computer storage media of claim 12, theinstructions further cause the one or more computing devices to:determine an order of actions in the first sequence of actions;calculate a sum of complexity of the first sequence of actions; andsearch the repository based at least in part on the sum of complexity ofthe first sequence of actions and the order of actions.
 14. The one ormore computer storage media of claim 12, the instructions further causethe one or more computing devices to: measure the similarity measurementbetween the first sequence of actions and the second sequence of actionsbased at least in part on respective complexity scores assigned toactions in the first sequence of actions and the second sequence ofactions, or an editing distance to transform the first sequence ofactions to the second sequence of actions.
 15. The one or more computerstorage media of claim 12, the instructions further cause the one ormore computing devices to: determine a complexity measurement of thefirst sequence of actions being satisfactory to a complexity thresholdfor creating new macros.
 16. The one or more computer storage media ofclaim 15, the instructions further cause the one or more computingdevices to: retrieve predetermined respective complexity scores ofactions in the first sequence of actions.
 17. The one or more computerstorage media of claim 15, the instructions further cause the one ormore computing devices to: dynamically determine a complexity score ofan action in the first sequence of actions based at least in part on aduration to perform the action by the user or another user.
 18. Asystem, comprising: means for detecting a first sequence of actionsrelated to a user on a computing system; means for searching arepository of sequences of actions for a second sequence of actions thatis similar to the first sequence of actions based at least in part on acharacteristic of the first sequence of actions; means for determining asimilarity measurement between the first sequence of actions and thesecond sequence of actions; and means for automatically generating amacro based at least in part on the first sequence of actions and thesimilarity measurement.
 19. The system of claim 18, wherein thecharacteristic of the first sequence of actions comprises a complexitymeasurement of an action in the sequence of actions.
 20. The system ofclaim 18, wherein the similarity measurement is determined based atleast in part on an order of actions in the sequence of actions.